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ABSTRACT 



Historically, data have been collected from survey 
participants through a paper-and-pencil questionnaire or through interviews 
in person or on the telephone. This study compares the use of a new approach, 
interactive telephone data collection ( ITDCT) to traditional paper-and-pencil 
collection. ITDCT administers survey items through a digitized human voice 
using a computer software program. Respondents enter their responses by 
pushing telephone keypad numbers. From an initial sample of 5,908 students, 
data about the community and campus environment were collected from 867 
undergraduates though a questionnaire and from 666 through ITDCT. The overall 
response rate was relatively low because of the timing (last week of the 
semester) and some problems with the ITDCT system. Although there were 
significant differences between the two survey types, when the data were 
inspected by plotting the means and standard deviation, one could see the 
statistical differences as a result of the large sample size and the power of 
the tests- -small degrees of freedom in the numerator with large degrees of 
freedom in the denominator. No patterns suggested that social context effects 
played a role in item responses by survey type. Findings were not sufficient 
to demonstrate whether ITDCT is really a good substitute for a 
paper-and-pencil questionnaire, but they do suggest the usefulness of the 
technique for a short questionnaire. (Contains 2 tables, 11 figures, and 12 
references.) (SLD) 



******************************************************************************** 

* Reproductions supplied by EDRS are the best that can be made * 

* from the original document. * 

******************************************************************************** 








U.S. DEPARTMENT OF EDUCATION 

r~*4. Drto aq rr* K onH ImnrnuAmS 



PERMISSION TO REPRODUCE AND 
DISSEMINATE THIS MATERIAL 



CO 



□ Minor changes have been made to 
improve reproduction quality. 



received from the person or organization < 
originating it. 




\M iik . 



HAS BEEN GRANTED BY 



00 

O 



• Points of view or opinions stated in this 
document do not necessarily represent 
official OERI position or policy. 



TO THE EDUCATIONAL RESOURCES 
INFORMATION CENTER (ERIC) 



Research Methodology in the Information Age: 
A Comparison of Two Survey Techniques 



Judith A. Ouimet 
Gary R. Hanson 
March 24, 1997 

Presented at AERA 
Chicago, EL 



BEST COPY AVAILABLE 



By 




2 



Research methodology in the information age: A comparison of two survey techniques 



Survey research started in the 1930's when the "big three" pioneer commercial 
polling organizations went door to door gathering data to predict presidential elections, 
women’s rights, or other timely topics of interest (Glenn, 1995). Seventy years later, as 
we enter the 21st century, technology has propelled survey research into new arenas of data 
collection. What was once a tedious, labor intensive process with various interviewer and 
responder effects has now become more user friendly entailing fewer biases. 

One of the most important innovations for telephone surveys was random digit 
dialing. This technology increased the effectiveness of telephone sampling and 
implementation. Shangraw (1986, p. 108) reported, "The University of Wisconsin's 
Survey Research Laboratory estimated that an experienced computer assisted telephone 
interview (CATI) interviewer's productivity may be as much as 20% higher than that of 
the same interviewer using traditional survey methods." This technology, along with 
various other changes in survey research, will continue to change the way universities, 
researchers, and marketers design, implement, and analyze surveys. 

Historically, we have collected data from survey participants using one of two 
primary methods. We can ask survey participants to complete a paper and pencil 
questionnaire or we can conduct interviews in one of two ways: (1) face-to-face or (2) over 
the telephone using trained telephone survey interviewers (Bradbum & Sudman, 1982). In 
recent years, three various techniques have been employed to effectively collect survey 
responses: (a) computer-administered questioning (CAQ)— respondents directly answer 
questions posed on computer monitors (Liefeld, 1988, p.405); (b) computer assisted 
personal interviewing (CAPI) or computer assisted telephone interview (CATI)-the 
computer randomly dials the phone number and leads the interviewer through the survey 
questions, then the interviewer enters the responses directly into the computer; and, (c) 
automatic dialing and announcing device (ADAD)-completely automated system where the 



computer dials the preselected phone numbers and tapes the responses to each question 
thus eliminating the need for an interviewer (Dickinson et al., 1990, p. 28). 

Only recently, has the technology for a fourth survey data collection technique 
emerged. The interactive telephone data collection technique (ITDCT) was developed to 
administer survey items using a digitized human voice, through a computer software 
program (Hanson & Ouimet, 1994). The digitized voice asks the respondents a question 
and then provides a list of possible coded responses. The survey participants indicate their 
response to each survey question by pushing one of the numerical telephone numbers that 
correspond to a coded response. For example, press a "1" if your answer is yes or press a 
"2" if your answer is no. The response is directly entered into a flat file, thus eliminating 
the need for data entry. The ITDCT is not limited to pre-coded data. Open-ended 
questions can be administered by leaving a voice mail message. 

As with all new techniques, one has to look at the usefulness and practicality of this 
innovative type of system. In addition, the validity of this technique has to be explored. 
Questions such as what type of item questions and responses are conducive to the ITDCT 
method? Do people respond differently using the ITDCT then the traditional paper pencil 
technique? Is the ITDCT a suitable substitute for the traditional paper-pencil technique? 

All of these questions need to be addressed before researchers adopt the present technique. 

A brief overview of the ITDCT system follows. This overview contains an 
explanation of how the system works, as well as some pros and cons on the issue, and 
finally, the administrative advantages and disadvantages one might expect. 

Bradbum (1983, p. 289) and Kiesler (1990, p. 404) suggest that one way to 
determine the comparability issue of survey methods is to look at response effects. 
Response effects occur when participants refuse to answer certain questions, give 
incomplete responses, do not follow directions, yea-say, under report or over report 
socially sensitive information, and choose neutral categories. The last two are categorized 
as social context effects. The more disconnected a participant is from personal interaction 



with the interviewer, survey researcher, etc., the more likely the social context effect will 
be reduced thus, decreasing the response effect. 

There are many considerations to take into account when assessing the utility of the 
ITDCT. Among them are 1) comparability to other methods of survey administration, 2) 
ease of participation, and 3) economic advantages. For the purposes of this paper, the 
traditional paper-pencil survey method is used for the comparison. 

The ITDCT has similar qualities to that of the paper-pencil instrument in that both 
are self-administered questionnaires. In addition, both offer a relatively anonymous 
response environment which reduce the phenomenon of social context effects. Finally, 
both techniques lack the opportunity for the respondent to request question clarification. 

Kiesler & Sproull believe "The electronic survey is also unique because of the 
degree to which it lacks social context information (1986, p.405)." Their research suggests 
that electronic surveys make a research setting more impersonal and anonymous and, in so 
doing, reduces social context effects. As a result, Kiesler & Sproull support the theory that 
survey subjects become more "self-centered and relatively unconcerned with social norms 
as well as the impressions they give others" (1986, p. 405). 

The ITDCT is designed to emulate an electronic survey, which can reduce social 
context effects. Thus, there is the belief that the social context information can be 
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controlled using the ITDCT. The ITDCT respondents have been hypothesized as likely to 
give item responses that are unaffected by either interviewer effects or the participant's 
desires to give a socially acceptable response rather than reveal true responses. 

The ITDCT is designed to permit the respondent to answer questions as if they 
were self-administered. This provides a venue similar to that of the paper-pencil survey in 
that there is a constant and unwavering presentation of questions, thus eliminating 
interviewer effects; however, there is no opportunity for item clarification. Since there is 
no opportunity to clarify an unclear question the participant must consider the literal 



interpretation of all questions in any self-administered anonymous survey method (Havice, 

p. 100). 

An important consideration when implementing surveys is the ease of participation. 
Filling out surveys and returning them can be perceived as a nuisance. Phone solicitation 
can also be annoying and inconvenient. With the ITDCT , however, the participant initiates 
the call when s/he has the time or inclination to do so. This method of data collection will 
become more popular as people become more acclimated to getting information using 
automated telephone technology— i.e. retrieving checking account balances, flight arrivals 
and departures, etc., responding to questionnaires will be the next logical progression. 

The primary economic advantage is cost. In the education environment, if the 
institution is set-up with a computerized registration system, then the initial costs will be 
minimum. The savings accrued on printing costs, return postage, data entry and 
correction, and data preparation will help counter start-up costs. The disadvantages pertain 
to the format restrictions, initial development and start-up costs if hardware is not in place, 
and technical problems. 

ITDCT Mechanics 

To understand better how the ITDCT works, see Figure 1. Essentially, the survey 
questions and responses are digitized and a translation table is created. ITDCT requires the 
respondent to provide answers by pressing the numbers on their touch-tone telephone 
keypad. During the study, respondents with dial telephones or those who do not have 
telephones were instructed to use campus phones. The other steps are very similar to other 
self-administered survey. The main difference is in the data capture step. In the ITDCT 
method, the data are captured instantaneously and placed on a flat file to be analyzed at 
anytime. In the mail survey, more time is spent in data entry than in any other aspect of the 
survey process. 
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ITDCT Advantages 

In the ITDCT, responses are entered via the telephone keypad and are automatically 
entered into a computer file, eliminating the need for data entry. This streamlining results 
in a reduction of data entry error because the data are entered in by the respondent. This is 
accomplished by having the respondent press the number directly; however, if the number 
is not within the response, the ITDCT will restate the question prompting the respondent to 
re-select a response. Another benefit of this type of data collection is that it permits data to 
be immediately stored in a form which is easily and quickly assessable to statistical 
analyses thus decreasing time needed to accomplish the analysis and reporting. 

The ITDCT provides a more systematic way to respond to questions. The 
respondent is led through one question at a time, instead of being overwhelmed with 
numerous questions. This method can insure that all items are in fact answered because 
the program does not proceed to the next question until the previous one has been 
answered. There is, however, the option of skipping a question by pressing 'O’. 

Complex branching is another important feature of this system. The ITDCT will 
automatically administer the proper question with regard to the previous response. For 
example, the question "Have you ever been harassed or discriminated against due to your 
gender? If the responds was 'yes', then questions pertaining to the type, place, form, 
etc., would be asked; however, if the responds was 'no', then the ITDCT branched to 
another topic question. Using the ITDCT assures the removal of interview effects due to 
the lack of non-verbal communication between the respondent and interviewer. 

ITDCT Disadvantages 

As with all survey methodologies, there are drawbacks to this method of data 
collection as well. First, there are format restrictions. Consistency in the scale order, 
either descending or ascending, is imperative when a Likert scale is used. The response 
categories or choices need to be short and concise. For instance, the first question on the 
ITDCT survey originally asked participants which definition of community best described 
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their image of campus community. The definition choices were so long that discerning one 
definition from another was difficult. This was discovered in the pilot test when subjects 
informed us that the question was very tedious and difficult to follow. The fear that 
subjects would become uninterested and hang-up, resulted in that item being dropped from 
the ITDCT survey and left on the mail survey. 

The second restriction, closely related to the first, is the ITDCT does not permit 
respondents to edit their answers after they respond to all items. A respondent could 
change his/her response only by calling back and having the ITDCT re-administer the 
survey. A call back results in all previous responses being overwritten with the new. 

While this insures that one subject does not bias the results with multiple responses, the 
responses may be quite different from the original. 

The third restriction is the survey length. Respondents must listen and respond to 
the questions, which takes longer than reading and responding. Therefore, the total 
number of questions that can be asked is limited. 

The fourth restriction is the developmental costs. The initial outlay of purchasing 
the hardware and having the software designed for the specific hardware, can be quite 
expensive and a major deterrent to implementing this type of data collection. For instance, 
we had 30 people call exactly at the same time to see what would happen to the system; 
none of the subjects had any problems getting through. Unfortunately, the first day major 
glitches occurred resulting in many technical obstacles, which needed to be addressed and 
fixed. 

Method 

Sample Demographics 

Data were collected from a stratified random sample of 5,908 currently enrolled 
male and female undergraduate students during the 1993-1994 academic year. The sample 
was then divided into two sub samples: (1) traditional paper-pencil survey (N= 1,968) and, 
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(2) interactive telephone data collection technique (ITDCT) (N=3940). A total of 1,533 
students participated in the survey (26%). 

The target paper-pencil sample included approximately 200 students in each of the 
following ethnic-gender categories: White, African-American, Hispanic, Asian-American, 
and foreign. A total of 867 students returned the paper-pencil survey (44%). The ethnic- 
gender (male/female) breakdown was: White-85/100, African-American-53/85, Hispanic- 
82/98, Asian-87/106, and International- 102/69. 

The ITDCT survey included approximately 800 students in each of the same ethnic 
categories. A total of 666 students responded to the telephone survey with a 17% response 
rate. The ethnic breakdown for the telephone survey was: White-75/100, African- 
American-45/67, Hispanic-60/77, Asian- American-6 1/72, and Foreign-67/42. 

The low response rate was attributable to both the time of year the survey was 
administered and problems with the ITDCT. The survey was disseminated the last week of 
the semester and during the final exam period. Due to this busy time the response rate was 
lower than usual for the paper-pencil survey. The even lower response rate for the ITDCT 
survey was confounded by the systems hardware and software problems. 

Insert Table 1 about here 

Given the low response rate to the surveys, we wanted to compare the responders 
to the overall university population. Suskie (1992) suggested that a quality representation 
is more important than a high yield rate that is disproportional to the underlying population. 
Gender, race/ethnicity, and college will be used for comparison and will be described. 

In table 1 , we compared the survey response rates to the University's 
undergraduate population. Women responded at a higher rate than did men. Of the 
students who responded to the surveys, 46 to 47% were males and 53 to 54% were 
females, which is the opposite of the institution's population. The study sample was 
representative in respect to class level with no differences larger than 5%. As a result of 
the stratification, the sample was not representative of the underlying student population 
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with respect to ethnicity. In the mail survey, each ethnic group had at least a 20% 
representation except for the African Americans who were underrepresented by 4%; 
however, the ITD.CT had fewer International students responding. Overall, the participants 
were representative of the undergraduate population by college with two exceptions. In 
the ITDCT sample, engineering students responded at higher rate (21%) than are 
representative of the population (14%) and liberal arts majors responded at a lower rate 
(24% responded, 29% in population). 

Measures 

We used selected items from a survey developed at The University of Minnesota 
(Harrold, Hendel, Melton and Scouten, 1991) to measure students’ definitions of 
community, the importance of and the extent to which they experienced community, and 
the experiences of gender and racial discrimination and harassment on campus. 

The campus community survey consisted of two parts: 1) The University as a 
Welcoming Environment, and 2) Campus Diversity Experiences. A total of 37 possible 
questions 1 with multiple responses were presented. Seven questions, each were asked in 
each of the following categories: community experiences, satisfaction of community, 
student organization memberships, time spent on campus outside of class, approachability 
of faculty, and gender and racial discrimination or harassment. An open-ended question to 
provide an opportunity for students to offer suggestions on how to improve diversity and 
campus community was also asked. 

Data Collection 

A three-stage mailing was conducted, as suggested by Dillman (1978), between 
November 1993 and January 1994. Each mail survey included a postage paid return 
envelop for convenience. A cover letter on University letterhead was attached to the survey 



1 In the telephone survey these questions were in the form of dichotomous questions, which actually 
increased the total possible questions to 72. 



informing each student the survey's nature and purpose. A reminder letter was sent the 
first week of December to increase the response rate. In mid-January a third reminder letter 
and a second copy of the survey was mailed. 

For the ITDCT survey a three-stage mailing was used; however, enclosed in the 
first mailing was a letter of invitation to participate in the ITDCT technique. The letter had 
specific instructions as to how to access the ITDCT, a phone number for both on and off 
campus use, a survey-number, a unique-number which was used to collect demographic 
information, and rudimentary directions. As with the mail survey, a third letter was sent in 
mid- January to invite these students to participate in the ITDCT survey. 

Purpose 

The purpose of this study was to explore whether students responded differently by 
survey type. We were particularly interested in the following research questions: Are there 
gender, ethnic and class interactions across type of survey? Does "method" variance 
account for more of the item distribution variance than does gender, class, and ethnic 
variance? To answer these questions we used linear models, ANOVA, and mean 
comparisons. 

Results 

First, we applied a linear models approach utilizing the PROC REG procedure in 
SAS. The four attribute model had race, class, gender, and survey type. This model 
contained 1 1 parameters: race (White, Hispanic, Black) had three levels, class (freshmen, 
sophomore, junior, senior) had four levels, gender (male, female) and survey type (mail, 
telephone) each had two levels (or a 3 by 4 by 2 by 2 block ANOVA). We then used a 
TEST statement in the PROC REG procedure to test the following hypothesis: 

Students respond the same to paper-pencil and ITDCT survey items. 

Women respond the same as men respond to mail and ITDCT survey items. 
Men respond the same to paper-pencil and ITDCT survey items. 



Ho 1: 
Ho 2a: 
Ho 2b: 
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Ho 3a: African Americans respond the same to paper-pencil items as they do to ITDCT 

survey items. 

Ho 3b: Hispanics respond the same to paper-pencil and ITDCT survey items. 

Ho 3c: Caucasians respond the same to paper-pencil and ITDCT survey items. 

Ho 4: All within class levels respond the same to paper-pencil and ITDCT survey 

items. 

Ho 5: All within group gender by ethnic respond the same to paper-pencil and ITDCT 

survey items. 

Findings 

In this section, comparisons between the paper-pencil instrument and the ITDCT 
are made on the basis of mean differences by item response, gender, race, class, and 
gender by race. 

Results of this study demonstrate significant differences (pc.Ol) between the two 
methods in some item responses. Due to the shear number of in the sample, significant 
differences will occur. Nevertheless, some variables did not demonstrate significant 
differences nor did they demonstrate practical significance. Table 2 shows the various 
statistical differences by race, gender, class, and survey type. As a result of these statistical 
findings, we decided to plot the means and standard deviations and to examine both pattern 
differences (statistical significance) and practical significance simultaneously.. 

Insert Table 2 about here 

We chose eight questions of various social context sensitivity and plotted the survey 
type means and standard deviation. To see if there are differences by gender, ethnic, 
class, and gender by ethnic we plotted the means of each group to ascertain if they fell 
within a .5 standard deviations interval above or below the overall item means. This plot 
provides a visual display mapping the relationship between the similarity of item response 
by survey type. Data in Figures 2 through 5 show that there really were not any practical 
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item response differences by survey type, although it is easy to see differences by gender, 
ethnic and class. 

A cursory look at the significant differences by item response provided no patterns 
with respect to social context effects. Two perceived socially sensitive questions 
concerning gender and racial harassment were asked. The mean for the gender harassment 
item (Figure 2 & 3) was statistically significant by survey type (F[l, 1480] = 1 1.05, p< 
.001) but the mean for the racial item (Figure 4 & 5) was not (telephone M=1.62, mail 
M=1.65). A quick look at figures 2 and 3 indicate there was virtually little survey type and 
item response difference; yet, statistically there was a difference. Other items that were 
considered less socially sensitive (see Figures 6 to 11) such as, experience community 
(F[l, 1480] = .95, NS), time spent on campus (F[l, 1480] = .21, NS), importance to 
experience a sense of community (F[l, 1480] = 48.76, p< .001), and satisfied with 
experiences (F[l, 1480] = 14.10, p< .001) were also found not to be practically significant 
by survey type. 

In Figure 8 and 9 one can see that Black and Hispanic females have slight item 
response differences (Black: telephone M=2.34, mail M=2.59; Hispanic: telephone 
M=2.51, mail M=2.72) than did White females (telephone M=2.47, mail M=2.45), these 
difference though, were not statistically significant. 

In the gender discrimination item, figures 2 and 3, which were statistically 
significant by survey type, there were slight practical significance's. Once again, Black 
females and Hispanic females responded differently (Black: telephone M=1.58, mail 
M=1.69; Hispanic: telephone M=1.71, mail M=1.79) than did the other gender by ethnic 
groups. There were also slight practical differences for men (telephone M=1.84, mail 
M=1.93) and for sophomores (telephone M=1.87, mail M=1.74). 

We also looked at the R-square by item for the full model (race, gender, class, and 
survey type). For five of the eight questions, the R-square was less than .10 with the 
largest R-square value of .20 on the racial discrimination and harassment question. 
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Discussion 



The purpose of this study was to: 1) determine whether these self-administered 
methods produced similar responses and, 2) determine whether the ITDCT is a viable 
alternative to the self-administered paper-pencil technique. 

Significant differences between the two survey types emerged; however, when 
visually inspected by plotting the means and standard deviation, one could see the statistical 
differences were a result of the large sample size and the power of the tests-small degrees 
of freedom in.the numerator with large degrees of freedom in the denominator. 

When we compared the differences by item sensitivity we found no patterns that 
would suggest social context effects played a role in the item responses by survey type. 
Only one of the two questions that dealt with discrimination and harassment was 
statistically significant; yet, if one compared the means and standard deviations of each 
question, the practical significance was minimal. Recall also due to the number of tests 
conducted, it is possible that significant results can emerge by chance in this context (Type 
I error). 

We also wanted to know if there were response differences within gender, ethnic, 
and class level. For brevity , we included only four item responses. Figures 2 through 4 
show the different item means by overall, gender, race, and class. One can quickly see the 
mean differences within each category are less than .5 standard deviations from the overall 
mean supporting the hypothesis of no difference by survey type. 

We also looked at the statistical differences to see if there were survey type pattern 
differences. We saw no consistent patterns. The differences seemed to be erratic, flipping 
from one question to another. Kiesler and Sproull had similar findings when they 
compared electronic survey responses to those administered in an equivalent paper-pencil 
method (1986, p 403-404). They predicted specific differences between these instruments, 
but their research found ". . . considerable similarity of response between the paper and 
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electronic survey but not so much that the two may be considered interchangeable without 
further research" (1986, p. 402-413). 

In contrast to Kiesler and Sproull (1986), we had hypothesized there were no 
practical or significant differences across survey type. The validity of this finding, with 
less than a 17% response rate for the ITDCT, even though we had a sample of 666, must 
be questioned. It is difficult to determine whether these two methods really do produce 
similar responses. We suggest further studies be implemented to explore the usefulness 
and reliability of the ITDCT. 

The question of whether the ITDCT is a viable alternative to the self-administered 

paper-pencil technique is two-fold. First, the results from this study appear to illustrate that 

{ 

social context effects do not differ from one self-administered instrument to the other. We 
did not see any consistent differences across survey type which suggests an absence of 
social context effects. Kiesler and Sproull (1986) "espoused that self-administered 
questionnaires are relatively anonymous and tend to reduce respondent's concern of over 
presenting themselves in a good light (p.405)." This is important to researchers when the 
information gathered is of a sensitive nature. Having another tool available, like the 
ITDCT, will facilitate research of this type. 

The second consideration, is the importance of item response construction for 
automated surveys. Havice and Banks (1990) found that for automated surveys careful 
construction was crucial. One reason for this is the lack of question clarification. Without 
the opportunity to clarify, a misinterpretation that could end in erroneous results might 
occur. This problem is not distinctive to the ITDCT, but is also a concern in the paper- 
pencil instrumentation as well. " It is also worth noting that a self-completion questionnaire 
does not avoid most of the sources of error such as questions being abbreviated, misread, 
omitted in error, or the routing may be incorrectly followed." (Berry O'Brien, 1978, p. 



A third consideration to the ITDCT is the necessity of keeping questions and 
responses short and clear. The longer the question, the more likely a participant is to not 
understand what is being asked. The effects of double-barreled questions, leading 
questions, or loaded questions would be enhanced because of the inherent confusing nature 
of those types of questions. 

Is the ITDCT a viable alternative to the self-administered paper-pencil instrument? As with 
many areas of study, the answer is it depends. As mentioned above, our findings are 
insufficient to truly determine whether this technique is a good substitute, however, we 
believe the ITDCT could be a useful tool in various situations. This technique would be an 
excellent tool for a short survey. One that has approximately 20 true/false or Likert scale 
responses and can be completed in no more than 10 minutes. The data could be down 
loaded and analyzed within a day after survey participation was completed. In fact, if the 
statistical code was written ahead of time, analyses could be run at any point during data 
capture. 

In times where research funds are decreasing, researcher must find innovative ways 
to streamline data collection techniques and more importantly reduce costs. One possible 
way this can be done is by using the ITDCT system. More research needs to be 
implemented to determine whether the ITDCT is a suitable and practical substitute for the 
present techniques of data collection. 

Questions for Future Studies 

As indicated in the research there are limitations to this new technique; many other 
studies will be required to determine its validity. To determine whether the ITDCT is, in 
fact, a viable addition to survey methodology further investigation is required. 




0 



14 



References 



Dickinson, J.R., Faria, A.J. , and Friesen, D. (199 ). Live vs. automated telephone 
interviewing. Marketing Research. 6(1), 28-33. 

Hanson, G. and Ouimet, J. (1994). A work shop on survey methods. Paper presented at 
AAHE Assessment Forum, Washington, D.C. 

Havice, M.J. (1989). How response rates compare for human and digitized phone surveys. 
Journalism Quarterly . 66 (Spring) 137-42. 

Havice, M.J. and Banks, (1991). Live and automated telephone surveys: A comparison of 
human interviewers and an automated technique. Journal of the Market Research Society . 
33 (October), 91-102. 

Harrold, Hendel, Melton, and Scouten, (1992). Campus Community and Diversity. 

Univesity of Minnesota. 

Glenn, N. (1995). Survey Research Handouts. 

Kiesler, S. and Sproull, L. (1986). Response effects in the electronic survey. Public Opinion 
Quarterly. 50 (Fall), 402-13. 

Liefeld, J.P. (1988). Response effects in computer-administered questioning. Journal of 
Marketing Research. 25 (November). 405-9. 

OBrien, T. and Dugdale, V. (1978). Questionnaire administration by computer. Journal of 
the Market Research Society . 20 (October) 228-37. 

Shangraw, R. Jr (1986). Telephone surveying with computers: Administrative methodological 
and research issues. Evaluation and Program Planning . 9, 107-1 11. 

Studmen, S. and Bradbum, N. (1982). Asking Questions. (1st ed.). San Francisco: Jossey- 
Bass. 

Suskie, L. (1992). Questionnaire survey research: What works . Tallahassee: Association for 
Institutional Research. 

17 




15 



o 

ERIC 



Table 1 

A Comparison of the mail and telephone respondents with the U-Texas Undergraduate 
Student Population 





Mail 

Responders 


Phone 

Responders 


UT 

Undergrad 




N 


% 


N 


% 


N 


% 


Gender 

Men 


409 


47% 


308 


46% 


18,509 


53% 


Women 


458 


53 


358 


54 


16,697 


47 


Class 














Freshmen 


191 


22 


151 


23 


7,217 


21 


Sophomore 


152 


17 


151 


23 


6,767 


19 


Junior 


204 


24 


147 


22 


7,710 


22 


Senior 


320 


37 


217 


33 


13,512 


38 


Race 














White 


185 


21 


175 


26 


23,673 


67 . 


African American 


138 


16 


112 


17 


1,449 


4 


Asian American 


193 


22 


133 


20 


3,558 


10 


Hispanic 


180 


21 


137 


22 


5,058 


15 


Foreign 


171 


20 


109 


16 


1,342 


4 


College 














Architecture 


5 


1 


1 


<1 


334 


1 


Bus Adm 


123 


14 


109 


16 


5,123 


15 


Communication 


67 


8 


56 


8 


2,912 


8 


Education 


33 


4 


22 


3 


1,844 


5 


Engineering 


144 


17 


140 


21 


5,091 


14 


Fine Art 


22 


2 


17 


2 


1,520 


4 


Liberal Arts 


229 


26 


157 


24 


10,277 


29 


Natural Science 


200 


23 


137 


21 


6,717 


19 


Nursing 


18 


2 


11 


2 


624 


2 


Pharmacy 


19 


2 


12 


2 


491 


1 


Social Work 


7 


1 


4 


<1 


273 


1 


Total 


867 


100% 


666 


100% 


35,206 


100% 
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Table £ 



ANOVA Probabilities and f-tests by race, gender, class, and survey type and R-square for the university as 
a welcoming environment and discrimination and harassment (N=1533) 







Race 




Gender 




Class 




Survey Type 


RSq. 




Question 


F 


Pr 


F 


Pr 


F 


Pr 


F 


Pr 




Q2 


Extent it is important to experience a 


3.47 


0.008 


9.62 


0.002 


1.75 


NS 


48.76 


0.001 


.09 




sense of community 




















Q3 


Experience community 


3.69 


0.005 


3.79 


0.005 


8.90 


0.001 


0.95 


NS 


.08 


Q4 


Satisfied with experiences 


5.93 


0.001 


1.47 


NS 


3.24 


0.022 


14.10 


0.001 


.09 


Q5 


Involved in student organizations 


8.73 


0.001 


2.77 


NS 


1.91 


NS 


4.91 


0.027 


.09 


Q6 


Time spent outside of class on campus 


17.37 


0.001 


3.82 


NS 


7.45 


0.001 


0.21 


NS 


.12 


Q7 


Approaching faculty* 


1.76 


NS 


1.83 


0.023 


4.11 


0.007 


16.78 


0.001 


.08 


Q8 


Gender discrimination/harassment 


1.86 


NS 


63.73 


0.001 


2.43 


0.064 


11.05 


0.001 


.10 


Q12 


Racial discrimination/harassment** 


53.06 


0.001 


0.14 


NS 


6.78 


0.001 


1.62 


NS 


.20 



•Question 7 had a race by survey type interaction (f-value 2.67 pr>0.031). 

••Question 12 had the following interactions. They are race by survey type (f-value 3.05 pr>0.016); gender 
by survey type (f-value 4.16 pr> 0.042); and race by gender by class by survey type (f-value 1.48 pr> 0.010). 
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Figure 2 



Gender Discrimination & Harassment 
Means by Survey Type by Race, Gender & Class 
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Figure 3 



Gender Discrimination & Harassment 
Means by Survey Type by Race & Class 
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Figure 4 



Racial/Ethnic Discrimination & Harassment 
Means by Survey Type by Race, Gender & Class 
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Figure 5 



Racial/Ethnic Discrimination & Harassment 
Means by Survey Type by Race & Gender 
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Figure 6 



Time Spend on Campus Outside the Classroom 
Means by Survey Type by Race, Gender & Class 
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Figure 7 



Time Spend on Campus Outside the Classroom 
Means by Survey Type by Race & Gender 
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Note: mean=3.33 sd=1.3 ' ' denotes mean - - denotes .5 sd above and below mean 
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Figure 8 



Experience Community 

Means by Survey Type by Race, Gender & Class 
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Experience Community 
Means by Survey Type by Race & Gender 
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Figure 10 



Student Satisfaction 

Means by Survey Type by Race, Gender & Class 




Note: mean— 3.45 sd— .99 ' ' denotes mean denotes .5 sd above and below mean 
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Figure 11 



Student Satisfaction 

Means by Survey Type by Race & Gender 
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Note, mean— 3.45 sd— .99 denotes mean denotes .5 sd above and below mean 
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